[Feature] Temporal Pipeline Parallelism & Stream Batch for Real-Time Video by mnasser02 · Pull Request #3099 · vllm-project/vllm-omni

mnasser02 · 2026-04-24T08:06:00Z

Purpose

Related issue: #2280.

Tested on Lingbot #3701 (not merged yet).

Implements:

SupportsMicroStepExecution on LingbotWorldFastPipeline with StreamVAE
micro-step execution path with async comms, double-buffering, compute/comm overlap
StreamBatchScheduler driving micro-step execution pipeline, with SLO-adaptive batch size

Not in scope:

motion-aware noise controller, adaptive sink tokens, RoPE offset reset
VAE-as-stage / run-time adaptive scheduling of transformer blocks across PP stages
Benchmarks vs SP baseline, PP×SP hybrid (SP not supported by Lingbot)

Test Plan

pytest -v \
tests/diffusion/test_diffusion_micro_step_pipeline.py \
tests/diffusion/test_diffusion_scheduler.py::TestStreamBatchScheduler

Test Result

Unit tests: 31 passed, 17 warnings
E2E on A100s:

Latency

Each cell: e2e_s / ttff_s (seconds).

A chunk is 12 pixel frames.

	num_chunks=1	4	8	12	16	20
GPUs=1	7.0s / 7.0s	32.9s / 24.6s	OOM	OOM	OOM	OOM
2	7.5s / 7.5s	21.2s / 16.3s	45.7s / 28.3s	OOM	OOM	OOM
3	8.0s / 7.9s	20.7s / 16.7s	38.9s / 26.5s	60.0s / 34.4s	OOM	OOM
4	8.4s / 8.4s	15.0s / 10.6s	32.7s / 21.9s	51.8s / 31.5s	72.3s / 39.2s	95.0s / 44.9s

Per-rank occupancy (PP=4, num_chunks=20)

Throughput vs GPU count

Micro-step latency distribution (PP=4, num_chunks=10)

PP=4, num_chunks=20

pp4_c20.mp4

Differences in the output from the original can be noticed since chunks can attend to previous chunks that aren't at their final denoising step.

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

hsliuustc0106 · 2026-04-24T10:13:03Z

Ready for full review when WIP status is removed. Preliminary scan available on request.

Note: Test plan and test results sections are currently empty. Please provide:

Test scripts & commands for the new pipeline parallelism and stream batch scheduler
E2e test results demonstrating the 40+ FPS target on 4×H100 with sub-500ms TTFF
Benchmark comparisons against SP baseline and PP×SP hybrid

Signed-off-by: Miguel Vieira Pereira <miguel.vpereira14@gmail.com> Implement Lingbot World Transformer into vllm-omni Signed-off-by: Miguel Vieira Pereira <miguel.vpereira14@gmail.com> Implement KV cache abstraction for Lingbot World Fast Add script to offline generation using Lingbot World Fast Implement online serving for Lingbot World and camera-based world models in general

Implement SupportsStepExecution protocol on Wan22Pipeline, decomposing the monolithic forward() into prepare_encode, denoise_step,step_scheduler, and post_decode. Add denoise_micro_step for temporal PP. Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

>> Different ranks work on different chunks. A context manager that views the req state of a rank as a chunk state allows benefitting from existing functionalities. Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

…ed task Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

… pipeline (B and T hardcoded for now) Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

…instead of sync send/recv Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Plain P2P on size-2 PG triggers lazy sub-comm creation that requires the peer present. Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Miguel Vieira Pereira <miguel.vpereira14@gmail.com>

Signed-off-by: Miguel Vieira Pereira <52176659+Miguel0312@users.noreply.github.com> Signed-off-by: Miguel Vieira Pereira <miguel.vpereira14@gmail.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Miguel Vieira Pereira <miguel.vpereira14@gmail.com>

asukaqaq-s · 2026-05-27T12:55:36Z

Hi, may I ask if there is an estimated plan for merging this PR? I have some features that need to be built on top of it.

mnasser02 · 2026-05-27T13:18:56Z

@asukaqaq-s @fhfuih

Currently, the core functionality of StreamDiffusionV2 (temporal PP through micro-step execution path and StreamBatchScheduler) should be almost ready. However, we can't test it properly since causal Wan isn't available in vllm-omni (in the paper, SDV2 builds on CausVid applied to Wan); not sure if this is being worked on, or this can be left for a future PR.

We have done some tests on a not-so-correct V2V adaptation of Wan2.1-T2V-1.3B to ensure functionality. I think that before the end of this week, we can open the PR (after a cleaning the code a bit) and share our preliminary results.

Update: we decided to test on Lingbot #3701

…ream-diffusion Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

chatgpt-codex-connector · 2026-05-30T00:09:27Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

hsliuustc0106 · 2026-05-30T00:24:16Z

resolve conflicts please

…diffusion Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

mnasser02 force-pushed the stream-diffusion branch from 1d341e8 to 4c8e2b7 Compare April 24, 2026 08:40

mnasser02 force-pushed the stream-diffusion branch from 8a0f16b to b34cf19 Compare April 27, 2026 11:52

wtomin mentioned this pull request May 7, 2026

[RFC]: vLLM-Omni Diffusion Module — Q2 2026 Roadmap #2226

Open

25 tasks

Miguel0312 and others added 26 commits May 18, 2026 14:10

remove denoise_micro_step

94141ec

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

add unit tests for wan22 step exec

e144462

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

add RankTask and track per-rank tasks in DiffusionSchedulerOutput

a542a6c

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

add num_chunks sampling param

a493fb3

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add execute_micro_step execution path that works on a per-rank assign…

d9cb798

…ed task Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add StreamBatchScheduler that controls the flow of chunks through the…

3414a6e

… pipeline (B and T hardcoded for now) Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add StreamBatchScheduler tests

778bc66

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add micro-step execution pipeline tests

95a2bd1

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add stream_batch arg

e18732d

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Set reply rank to 0

90a9bfe

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Fix blocking send/recv resulting in deadlock by registering metadata …

5374266

…instead of sync send/recv Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfix

864ccd1

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfix

18daf55

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfix

ea12e8b

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Implement fully async isend/recv_dicts and migrate PPMixin to use them

78d80e9

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Rmv stale code

986571c

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfix

f502e96

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Make sending dict schema unblocking

d111ca0

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add warmup for nccl comms on init

26e7b8e

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Use batch_isend_irecv instead of plain isend/recv

34a8002

Plain P2P on size-2 PG triggers lazy sub-comm creation that requires the peer present. Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Handle tensor receive as a task

49f5807

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Chunk-aware buffers

63bb687

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

fix

e77972d

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

amy-why-3459 mentioned this pull request May 26, 2026

[RFC]: World Model Support #1987

Open

19 tasks

Miguel0312 and others added 4 commits May 26, 2026 09:36

Implement tests for Lingbot World Fast

0109533

Signed-off-by: Miguel Vieira Pereira <miguel.vpereira14@gmail.com>

Merge branch 'main' into lingbot-fast

69f64e5

Signed-off-by: Miguel Vieira Pereira <52176659+Miguel0312@users.noreply.github.com> Signed-off-by: Miguel Vieira Pereira <miguel.vpereira14@gmail.com>

bugfixes

778e2ff

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Update documentation

b68064e

Signed-off-by: Miguel Vieira Pereira <miguel.vpereira14@gmail.com>

mnasser02 added 6 commits May 28, 2026 12:10

Merge branch 'lingbot-fast' of github.com:zzhang-fr/vllm-omni into st…

883270d

…ream-diffusion Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

revert wan changes

c7c2cf8

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

fixes

21a1cbd

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

add stream batch support to lingbot

234841c

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

update tests

10a5e40

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

add docs

7762df5

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

mnasser02 changed the title ~~[WIP][Feature] Temporal Pipeline Parallelism & Stream Batch for Real-Time Video~~ [Feature] Temporal Pipeline Parallelism & Stream Batch for Real-Time Video May 30, 2026

mnasser02 marked this pull request as ready for review May 30, 2026 00:09

mnasser02 requested review from Gaohan123, Isotr0py, RuixiangMa, SamitHuang, ZJY0516, david6666666, hsliuustc0106, princepride, tzhouam, wtomin, yenuo26 and ywang96 as code owners May 30, 2026 00:09

Merge branch 'main' of github.com:vllm-project/vllm-omni into stream-…

b8ff268

…diffusion Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Temporal Pipeline Parallelism & Stream Batch for Real-Time Video#3099

[Feature] Temporal Pipeline Parallelism & Stream Batch for Real-Time Video#3099
mnasser02 wants to merge 56 commits into
vllm-project:mainfrom
zzhang-fr:stream-diffusion

mnasser02 commented Apr 24, 2026 •

edited

Loading

Uh oh!

hsliuustc0106 commented Apr 24, 2026

Uh oh!

asukaqaq-s commented May 27, 2026

Uh oh!

mnasser02 commented May 27, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented May 30, 2026

Uh oh!

hsliuustc0106 commented May 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mnasser02 commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Latency

Per-rank occupancy (PP=4, num_chunks=20)

Throughput vs GPU count

Micro-step latency distribution (PP=4, num_chunks=10)

PP=4, num_chunks=20

Uh oh!

hsliuustc0106 commented Apr 24, 2026

Uh oh!

asukaqaq-s commented May 27, 2026

Uh oh!

mnasser02 commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented May 30, 2026

Uh oh!

hsliuustc0106 commented May 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mnasser02 commented Apr 24, 2026 •

edited

Loading

mnasser02 commented May 27, 2026 •

edited

Loading